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Abstract: 


The  vertebrate  retina  consists  of  a  series  of  sheets  of  neurons,  each  containing  a  patterned 
representation,  or  "neural  image"  corresponding  to  the  incoming  visual  scene.  Most  of  the  operations 
can  be  described  as  convolutions  implementing  small  kernels  involving  local  neighborhoods.  These 
patterns  become  more  complex  as  the  visual  signal  passes  through  the  retina  as  each  new  convolution 
builds  upon  previous  ones.  Because  we  have  access  only  to  single  cells,  these  patterns  of  activity  have 
never  been  recorded.  As  a  first  step  in  visualizing  these  patterns  we  have  attempted  to  simulate  them 
using  modem  image  processors  operating  in  real  time,  that,  like  the  retina  invoke  local  convolutions, 
in  an  effort  to  understand  the  mechanisms,  and  the  physiological  roles  of  each  of  these  biological 
image  processing  operations.  In  addition,  we  have  now  developed  methods  for  measuring  these 
patterns  in  living  retinas.  The  comparison  between  the  modeled  and  measured  patterns  will  show  us 
what  components  of  retinal  function  have  not  yet  been  incorporated  into  our  understanding,  as 
expressed  by  the  models.  These  studies  will  help  us  to  understand  biological  image  processing  by 
forcing  us  to  assemble  the  retina  using  well-defined  physiological  functional  building  blocks,  derived 
from  retinal  function.  A  long-term  objective  of  this  research  is  the  implementation  of  biological  image 
processing  algorithms  for  man-made  systems  such  as  tracking,  target  identification,  guidance  and 
navigation. 
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Computer  models  of  retinal  function 

•  Objectives: 

The  vertebrate  retina  processes  information  by  passing  the  visual  signal  through  a  series  of 
2 -dimensional  sheets  of  neurons  (1)  as  illustrated  in  the  caricature  shown  in  Figure  1 ,  At  each  retinal 
level  these  neuronal  anays  express  a  pattern  of  activity  the  “neural  image,"  that  reflects  the  image 

processing  operations  that  have  been 
performed.  Most  of  the  operations  can  be 
described  as  convolutions  implementing 
small  kernels  involving  local 
neighborhoods.  These  patterns  become 
more  complex  as  the  visual  signal  passes 
oifferenci  through  the  retina  as  each  new  convolution 
SROAOCASt  uM/NG^^ilds  upon  previous  ones.  We  have 
attempted  to  reproduce  these  patterns  of 
activity  using  modem  image  processors 
operating  in  real  time,  that  also  invoke  local 
convolutions,  in  an  effort  to  understand  the 
mechanisms,  and  the  physiological  roles  of 
Figure  I.  Schematic  of  vertebrate  retina  shoving  the  S  maln^oh  of  these  biological  image  processing 
layers  of  cells  identified  by  their  names  (left)  and  their /w»cr/<woperations.  These  Studies  will  help  us  to 
(right).  Convolutions  at  each  layer  perform  /mag«understand  biological  image  processing  by 
transformations  that  alter  the  pattern  of  activity  that  exists  at  each  fafag  us  assemble  the  retina  using 

retinal  level  well-defined  physiological  functional 

building  blocks,  derived  from  retinal 
function.  A  long-term  objective  of  this  research  is  the  implementation  of  biological  image 
processing  algorithms  for  man-made  systems  such  as  tracking,  target  identification,  guidance  and 
navigation. 


OUTPUT 


Methods: 


The  neural  images  generated  by  the  biological  retina  are  dynamic  2-dimensional  patterns 
formed  via  convolutions  based  upon  small  (neighborhood)  kernels.  These  kernels  dictate  in  what 
ways  the  activity  at  each  pixel  is  a  function  of  the  activity  of  adjacent  pixels  in  its  local  neighborhood. 
These  operations  are  best  and  most  easily  simulated  through  the  use  of  currently-available  image 
processing  computational  systems.  We  have  utilized  two  different  systems  for  studying  retinal 
processing.  Our  first  computational  system  was  the  POPE  (Pipelined  Image  Processing  Engine). 
This  was  a  digital  machine  operating  in  real  time  that  performed  local  convolutions.  More  recently 
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we  have  utilized  the  CNN  paradigm  (4,8),  a  somewhat  more  versatile  system  oprating  on  similar 
principles,  but  with  the  capability  of  being  implemented  in  analog  VLSI.  We  adapted  the  space 
and  time  constants  derived  from  physiological  single  cell  measurements  taken  from  the  retina  of 
the  salamander  to  form  the  appropriate  convolution  kernels,  synaptic  functions,  and  interactions 
mediating  retinal  function.  Both  of  these  systems  have  generated  patterns  of  activity  that  simulate 
the  image  processing  fiinctionsof  the  biological  retina  at  each  level  of  operation. 

Concurrently  with  the  development  of  the  computational  retinal  models,  we  have  been 
implementing  methods  for  actually  measuring  the  patterns  of  activity  generated  by  the  biological 
retinas.  The  objective  here  is  to  compare  the  patterns  generated  by  the  biological  retinas  with  those 
formed  on  the  image  processors.  Essentially,  we  are  comparing  the  measurements  with  our 
hypotheses  of  retinal  function.  To  the  extent  that  the  model  and  hypothesis  match,  we  can  be 
reasonably  satisfied  that  our  hypothesis,  to  the  level  that  it  reflects  retinal  function,  is  accurate.  More 
interestingly,  when  there  is  a  mismatch  between  measurement  and  hypothesis,  we  will  be  led  to  a 
new  area  of  retinal  functional  that  had  previously  been  obscure. 

So  far  in  this  work,  our  descriptions  of  outer  retinal  function  seem  adquat  e  because  we  have 
matched  the  measurements  taken  from  the  physiological  retina  with  the  patterns  generated  in  the 
model.  We  are  presently  working  on  the  measurement  of  inner  retinal  function  as  reflected  in  the 
behavior  of  the  output  cells,  die  ganglion  cells.  It  is  in  this  region  of  retinal  function,  mediating 
movement  detetion  and  directional  selectivity,  where  little  is  known  about  retinal  circuitry  or 
function,  that  we  expect  to  fmd  disparities  between  hypothesis  and  measurement. 

•  Results  during  the  grant  period 


Our  research  has  focused  upon 
implementing  as  much  of  the  image 
processing  capability  of  the  retina  as 
possible  in  the  model  system.  The  results 
of  this  work  is  embodied  in  Paper  #1 
included  in  Appendix  I  (7),  as  well  as  in  a 
videotape  that  has  been  widely  distributed 
to  the  academic  community  throughout 
the  world.  A  copy  of  this  tape  has  already 
been  sent  to  John  Tangney  at  AFOSR,  and 
a  copy  is  available  upon  request. 

The  parameters  we  used  to 
implement  the  overall  model  were  taken 
from  the  physiological  studies  of  the 
salamander,  perhaps  the  best-studied 
Figure  2.  Space  constants  for  retinal  networks  derived  from  ^vertebrate  retina  we  have  today.  To 
physiological  data  These  space  constants  are  implemented  ^illustrate.  Figure  2  shows  a  summary  of  the 
dijjuston  templates  at  each  level  Space  constants  incorporated  into  the 

model. 


Nonlinear  Synaptic  Functions 


PRESYNAPTIC  VOUAGE 
Figure  15 

Figure  3.  Synaptic Junctions  defining  the  relationship  of activity  In  each  retinal  cell  as  a  function  of  the  activity  of 
the  cells  that  drive  It.  Asa  general  rule,  cells  in  the  outer  retina  were  driven  by  continuous  functions  over  a  broad 
range  of  potentials,  whereas  cells  in  the  inner  retina  were  driven  by  much  steeper  functions  that  are  almost 
“threshold"  functions. 


Timing  of  retinal  signals  was  also  adapted  from  the  physiological  measurements,  and  the 
data  used  to  implement  this  timing  is  discussed  in  Paper  1 .  Finally,  the  synaptic  functions,  derived 
from  the  measured  input-output  relations  between  retinal  cells,  was  implemented  from  lookup  tables 
in  the  model.  A  series  of  these  synaptic  functions  is  illustrated  in  Figure  3. 

These  three  functional  relationships  defining  time,  space  and  relative  magnitude  of  signals, 
were  combined  according  to  the  roles  of  retinal  connectivity  as  well  as  we  know  them  today  (5),  to 
generate  die  model  of  retinal  function  described  in  Paper  I  and  illustrated  in  time  and  space  on  the 
videotape.  Some  of  the  highlights  of  these  studies  are  illustrated  below. 
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Figure  4.  Transformations  at  the  outer  retina  The  original  intensity  levels  are  compressed  by  adaptation 
mechanisms  in  each  photoreceptor  to  generate  a  series  of  compressed  response  levels.  These  levels  are  normalized 
with  respect  to  local  average  Intensities  by  the  action  of  the  horizontal  cell  network  These  locally  normalized  signals 
are  then  amplified  at  the  synapse  between  photoreceptors  and  bipolar  cells  to  emphasize  the  slope  of  the  signal  at 
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*  Outer  Retinal  Function 

The  first  major  interaction  in  the  retina  takes  place  at  the  photoreceptor  terminals  where  the 
broad  ranging  horizontal  cells  feed  back  to  the  photoreceptors  to  form  a  contrast-enhanced  version 


Figure  5.  Patterns  of activity  at  the  bipolar  cells.  A.  Activity  measured  in  a  bipolar  cel!  "array"  showing  the 
presence  of  some  inhibition  around  the  edges  of  the  square.  B.  Patterns  generated  in  the  model  using  lateral 
inhibition  from  horizontal  cells.  C.  Difference  between  the  measured  and  modeled  patterns  at  the  bipolar  cell 
array.  These  differences,  mainly  spike-Uke  projections  at  the  corners  of  the  responses  to  a  square ,  represent 
activity  that  was  not  accounted  for  in  the  model. 

of  the  original  image  at  the  photoreceptor  terminals  (2,6)  as  shown  in  Figure  4.  The  signals 
generated  across  the  array  ofbipolar  cells  in  response  to  a  square  patch  of  illumination  are  illustrated 
in  Figure  5.  This  shows  the  bipolar  cell  response  to  the  square  embedded  in  a  saucer  of  horizontal 
cell  inhibition.  A  snapshot  of  die  form  ofbipolar  cell  patterns  derived  from  both  the  hypothetical 
and  actual  physiologically-measured  results  are  shown,  along  with  a  pattern  showing  the  differences 
between  these  two  forms  of  activity  at  one  instant  of  time.  It  is  through  an  analysis  of  these 
differences  that  we  will  be  able  to  leam  about  retinal  function,  because  the  differences  are  due  to 
components  of  function  that  we  have  not  included  in  the  model.  For  example,  in  the  comparison  of 
Figure  S,  there  is  a  set  of  peaks  in  the  activity  of  the  model  that  don’t  exist  in  the  measurements. 
This  indicates  that  either  the  space  constants  for  the  horizontal  cell  or  bipolar  dendrites  was  in  error 
in  our  model  and  should  be  adjusted  based  upon  a  more  accurate  set  of  physiological  measurments. 
These  results  also  show  that  interactions  at  die  outer  retina  perform  more  of  a  gain-setting  than  an 
edge  enhancing  function.  It  would  require  a  much  narrower  space  constant  for  the  horizontal  cells 
to  mediate  an  edge-enhancing  function  (3). 

•  Activity  it  the  Inner  Plexiform  Layer 

The  inner  retina  mediates  functions  associated  with  timing  such  as  movement  detection, 
change  detection,  and  dynamic  contrast  gain  control.  These  functions  are  subserved  by  two  main 
populations  of  amacrine  cells  as  shown  in  Figure  6.  One  population,  generating  a  sustained  response, 
appears  to  transmit  inhibition  over  a  region  about  the  same  dimension  over  which  it  receives  its 
input,  as  shown  in  Figure  6A.  The  other  population  receives  input  locally,  then  broadcasts  its  output 
over  a  broad  area,  but  only  transiently,  as  shown  in  figure  6B.  These  cells  have  processes  that  can 
extend  1/3  of  die  way  across  the  retina,  and  their  inhibitory  activity  dominates  the  activity  of  all 
cells  that  send  signals  from  the  retina  to  higher  centers.  The  retinal  output  cells,  the  ganglion  cells, 
are  therefore  controlled  by  these  inhibitory  signals,  and  their  effects  are  superimposed  upon  the 
activity  of  the  bipolar  cells  shown  in  Figure  S,  as  illustrated  in  the  behaviors  of  the  ganglion  cell 
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Figure  6.  A.  Narrow  field  amacrine  cell,  probably  responsible  for  local  rather  than  lateral  inhibition.  Its  main  rote 
may  be  to  truncate  activity  in  local  regions  creating  transient  acti vity  In  cells  that  signal  only  the  arrival  or  departure 
of  visual  targets.  S .  Wide  field  amacrine  cells.  Their  function  is  to  receive  inputs  locally  (in)  and  then  to  broadcast 
activity  aver  wide  regions  of  the  retina  (out).  These  cells  may  be  involved  in  directional  selective  movement  detection 

mosaic  shown  in  Figure  7.  This  figure  shows  the  development  in  time  of  the  ganglion  cell  response. 

It  begins  with  a  square  area  of  activity  corresponding  to  the  excitatory  input  from  the  bipolar  cells 
in  A.  This  is  followed  by  a  trough  of  inhibition  that  begins  to  develop  in  B,  and  proceeds  to  spread 
across  die  retina  in  C  and  D.  As  this  trough  represents  the  activity  of  the  amacrine  cell  system 
described  in  B  of  the  previous  figure.  At  the  same  time,  a  ridge  of  activity  develops  around  the 
original  square  area  of  activity.  This  ridge  is  mediated  by  the  local  inhibition  of  the  narrow  field 
amacrine  cells  shown  in  part  A  of  Figure  6.  These  two  inhibitory  components,  represented  by  the 
ridge  and  trough,  are  the  simplest  manifestations  of  the  amacrine  cell  inhibition. 


Figure  7.  Development  in  time  of  the  ganglion  cell  response.  A.  Initial  response  showing  the  excitatory  input  from 
a  local  population  of  bipolar  cells  driven  by  the  square  stimulus.  B,  C.D.  Development,  in  time  of a  broad  inhibitory 
trough  mediated  by  the  wide  field  amacrine  cells,  as  well  as  a  narrower  ridge  of  inhibition  near  the  square  itself 
mediated  by  the  narrow  field  amacrine  cell  system. 

•  What  we’ve  learned  from  these  studies. 

The  modeling  exercise  presented  above  and  described  more  fully  in  Appendix  I  represents 
a  first  step  in  understanding  retinal  function  from  a  computational  point  of  view.  We  now  have  an 
assembly  of  functional  modules  embodied  in  the  model  descriptions  of  the  different  classes  of  retinal 
cell  networks.  Most  of  the  physiological  responses  can  be  approximated  by  these  components,  but 
the  actual  arrangement,  including  lateral  dimensions,  time  course,  synaptic  gains  and  weightings 


e 


can  only  be  determined  by  comparing  the  mode!  behavior  with  that  of  the  living  retina.  To  measure 
the  patterns  of  activity  in  the  actual  retina  we  have  constructed  a  multielectrode  recording  apparatus 
that  is  capable  of  measuring  the  coincident  activity  of  100  ganglion  cells.  This  will  provide  us  with 
information  about  correlative  activity  and  asymmetries  in  function.  In  addition,  we  have  developed 
methods  for  recording  with  patch  clamp  from  individual  ganglion  cells,  recordings  that  can  last  for 
hours.  By  moving  the  stimulus,  i.e.  a  square  box,  to  every  possible  position  with  respect  to  the 
ganglion  cell,  we  can  use  that  cell  as  a  representative  of  all  ganglion  cells  in  a  grid  that  would  have 
responded  to  the  stimulus.  These  recordings,  displaying  both  excitation  and  inhibition,  can  then  be 
played  back  in  sequence  to  provide  a  pattern  of  activity  which  we  can  compare  with  the  model. 
These  studies  will  give  us  a  clearer  indication  of  the  dimensions  and  time  course  of  the  retinal 
networks  underlying  different  forms  of  lateral  inhibition,  networks  that  have  never  been  studied 
before.  Thus,  the  model  provides  us  with  a  prototype  of  retinal  function  as  we  know  it  today,  which 
may  be  useful  for  generating  biologically-based  image  processing  system.  But  the  modeling  also 
leads  us  to  new  advances  in  our  understanding  of  the  biology  by  providing  valuable  analytical  tools 
that  have  never  before  been  available  to  biologists. 
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